System and method for hybrid kernel- and user-space incremental and full checkpointing

ABSTRACT

A system includes a multi-process application that runs. A multi-process application runs on primary hosts and is checkpointed by a checkpointer comprised of at least one of a kernel-mode checkpointer module and one or more user-space interceptors providing at least one of barrier synchronization, checkpointing thread, resource flushing, and an application virtualization space. Checkpoints may be written to storage and the application restored from said stored checkpoint at a later time. Checkpointing may be incremental using Page Table Entry (PTE) pages and Virtual Memory Areas (VMA) information. Checkpointing is transparent to the application and requires no modification to the application, operating system, networking stack or libraries. In an alternate embodiment the kernel-mode checkpointer is built into the kernel.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/167,481 filed May 27, 2016 titled SYSTEM AND METHOD FOR HYBRIDKERNEL- AND USER-SPACE INCREMENTAL AND FULL CHECKPOINTING, now issuedU.S. Pat. No. 10,019,327, issued on Jul. 10, 2018, which is acontinuation of U.S. application Ser. No. 14/292,156 filed May 30, 2014titled SYSTEM AND METHOD FOR HYBRID KERNEL- AND USER-SPACE INCREMENTALAND FULL CHECKPOINTING, now issued U.S. Pat. No. 9,354,977, issued onMay 31, 2016, which is a continuation-in part and claims priority fromU.S. application Ser. No. 13/096,461 filed Jan. 31, 2011 titled SYSTEMAND METHOD FOR HYBRID KERNEL- AND USER-SPACE CHECKPOINTING, now issuedU.S. Pat. No. 8,745,442, issued on Jun. 3, 2014, and is related tocommonly assigned: U.S. patent application Ser. No. 13/920,889 filedJun. 18, 2013 titled SYSTEM AND METHOD FOR HYBRID KERNEL- AND USER-SPACECHECKPOINTING USING A CHARACTER DEVICE, now issued U.S. Pat. No.9,164,843, issued on Oct. 20, 2015, and is related to U.S. patentapplication Ser. No. 12/334,660, filed Dec. 15, 2008, titled METHOD ANDSYSTEM FOR PROVIDING CHECKPOINTING TO WINDOWS APPLICATION GROUPS, nowissued U.S. Pat. No. 9,286,109, issued on Mar. 15, 2016, the disclosureof which is hereby incorporated by reference herein in its entirety. Thepresent invention is also related to commonly assigned U.S. patentapplication Ser. No. 12/334,634 filed Dec. 15, 2008 titled METHOD ANDSYSTEM FOR PROVIDING COORDINATED CHECKPOINTING TO A GROUP OF INDEPENDENTCOMPUTER APPLICATIONS, now issued U.S. Pat. No. 8,078,910, issued onDec. 13, 2011, the disclosure of which is hereby incorporated byreference herein in its entirety

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not Applicable

INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC

Not Applicable

NOTICE OF MATERIAL SUBJECT TO COPYRIGHT PROTECTION

A portion of the material in this patent document is subject tocopyright protection under the copyright laws of the United States andof other countries. The owner of the copyright rights has no objectionto the facsimile reproduction by anyone of the patent document or thepatent disclosure, as it appears in the United States Patent andTrademark Office publicly available file or records, but otherwisereserves all copyright rights whatsoever. The copyright owner does nothereby waive any of its rights to have this patent document maintainedin secrecy, including without limitation its rights pursuant to 37C.F.R. § 1.14.

BACKGROUND OF THE INVENTION 1. Field of the Invention

This invention pertains to software-based checkpointing of applicationsrunning on computer systems, computer networks, telecommunicationssystems, embedded computer systems, wireless devices such as cell phonesand PDAs, and more particularly to methods, systems and procedures(i.e., programming) for checkpointing and checkpoint-restoration ofapplications where the core checkpointing service is performed as akernel service.

2. Description of Related Art

In many environments one of the most important features is to ensurethat a running application continues to run even in the event of one ormore system or software faults. Mission critical systems intelecommunications, military, financial and embedded applications mustcontinue to provide their services even in the event of hardware orsoftware faults. The autopilot on an airplane is designed to continue tooperate even if some of the computer and instrumentation is damaged; the911 emergency phone system is designed to operate even if the main phonesystem if severely damaged, and stock exchanges deploy software thatkeep the exchange running even if some of the routers and servers godown. Today, the same expectations of “fault-free” operations are beingplaced on commodity computer systems and standard applications.

Checkpointing is a general technique used to capture some or all of anapplication's state and preserve the state for use at a later time. Theapplication state can, by way of example, be used to recovery a crashedapplication and to migrate, i.e. move, an application from one server toanother.

TODO: The present invention builds on the teachings in U.S. patentapplication Ser. No. 13/096,461 wherein Havemose (“Havemose”) teachesSYSTEM AND METHOD FOR HYBRID KERNEL- AND USER-SPACE CHECKPOINTING. Thepresent invention further builds on the teachings in U.S. patentapplication Ser. No. 13/920,889 wherein Havemose (“Havemose”) teachesSYSTEM AND METHOD FOR HYBRID KERNEL- AND USER-SPACE CHECKPOINTING USINGA CHARACTER DEVICE. In these two patent applications Havemose teachessystem and methods for checkpointing multi-process applications using ahybrid kernel-mode and user-mode checkpointer and the use of acharacter-device for checkpointing. In U.S. Pat. No. 7,293,200 Neary etal (Neary) disclose “Method and system for providing transparentincremental and multiprocess checkpoint to computer applications”. InU.S. patent application Ser. No. 12/334,660 Backensto et al (Backensto)teach METHOD AND SYSTEM FOR PROVIDING CHECKPOINTING TO WINDOWSAPPLICATION GROUPS providing similar checkpointing services to Windowsapplications, and in Ser. No. 12/334,634 Havemose (Havemose) teachMETHOD AND SYSTEM FOR PROVIDING COORDINATED CHECKPOINTING TO A GROUP OFINDEPENDENT COMPUTER APPLICATIONS. Neary, Havemose and Backensto use auser-space checkpointer combined with interception and functionality toadjust links to libraries and files for checkpoint restore.

OpenVZ (http://en.wikipedia.org/wiki/OpenVZ) approaches checkpointingdifferently by providing checkpointing using a custom kernel. In otherwords, checkpointing is provided using a custom operating system.

Virtual Machine technology, such as VMware®, XEN®, and KVM, offersimilar features, often using terminology such as snapshot and livemigration. Virtual machine technology, however, is an entire additionalsoftware layer sitting under the operating system, which adds overheadand management complexity.

The prior art thus requires functionality running in user space (Neary,Havemose and Backensto), a custom operating system (OpenVZ), or acommitment to a hardware virtualization platform (VMware, XEN and KVM).Having a checkpointer with extensive user space components makes thecheckpointer very dependent on system libraries and requires constantupdating as user libraries change between releases of the operatingsystem. Relying on a custom operating system requires application to becustomized for the custom operating system, which can reduce the numberof applications available to customers. Finally, a commitment tohardware/system virtualization can be expensive and change thedeployment and management model of applications.

There is therefore a need for a checkpointing service that runs fullytransparent to the applications, runs on standard operating systems, andoperates without requiring a hardware virtualization layer. The presentinvention provides checkpointing as a kernel service generally loaded asa loadable kernel module working along with user-space interceptors. Thekernel service may be dynamically loaded into the kernel and providescheckpointing services universally to all applications without requiringany application customizations or customization to the underlyingoperation system. Interceptors are loaded as part of loading theapplication. The kernel module checkpointer can be further optimized bymodifying the kernel. Likewise, no hardware or system virtualizationtechnology is required which keeps memory and hardware requirementsminimal.

BRIEF SUMMARY OF THE INVENTION

The present invention provides systems and methods forapplication-checkpointing that requires no modification to theapplications being checkpointed (i.e. is transparent) and works oncommodity operating system and hardware. The terms “checkpointer”,“checkpointing”, “taking a checkpoint” and “checkpointing service” areutilized herein interchangeably to designate a set of services whichcapture the some or all state of an application and store all or some ofthe application state locally or remotely, and “restoring” or“checkpoint-restore” designates the process of restoring the state ofthe application from said stored application checkpoint. The terms“checkpoint file” or “checkpoint” are utilized herein interchangeably todesignate the data captured by the checkpointing service. Generally, thecheckpoint files are written to local storage, remote storage, networkstorage or memory. In the present invention, the elements of thecheckpointing services are provided as a Loadable Kernel Module (LKM).Loadable kernel modules are also called Kernel Loadable Modules (KLM),kernel extensions, or simply Kernel Modules. Throughout the rest of thisdocument we use the terms Loadable Kernel Module and kernel moduleinterchangeably to designate the functionality across all operatingsystems.

When using checkpointing to move an application from one server (theprimary) to another server (the backup) the following terminology isused:

The terms “primary” and “primary application” are used interchangeablyto designate the primary application running on the primary host. Thehost on which the primary application is running is referred to as the“primary server”, “primary host” or simply the “host” when the contextis clear. The term “on the primary” is used to designate an operation oractivity related to the primary application on the primary server.

Similarly, the terms “backup” and “backup application” are usedinterchangeably to designate a backup application running on a backuphost. The host on which the backup application is running is referred toas a “backup server”, a “backup host” or simply a “host” when thecontext is clear. The terms “on the backup” or “on a backup” are usedinterchangeably to designate an operation or activity related to abackup application on a backup server.

The term “Live Migration” is used to designate the processes of moving arunning application or a running virtual machine from a primary serverto a backup server. The “migration” is “live” as the application is keptrunning for the majority of the move. Generally, live migration of bothapplications and virtual machines are planned; i.e. they are triggeredin response to an event. The event may be an operator choosing tomigrate the application/VM, a memory threshold being met, or otherpre-defined scriptable event. For the live migration to succeed both theprimary and the backup must operate during the entire live migrationprocess.

The term “fault” is used to designate an abnormal condition or defect ofa component, software, sub-system or equipment. Examples of faultsinclude a power supply burning out, a CPU overheating, and a softwarebug that crashes an application. Faults can happen at any time and arethus non-deterministic, i.e. unpredictable. The term “Fault Detection”designates the mechanism used to detect that a fault has occurred. Faultdetection is well known in the art and is therefore not further definedherein.

The following terms are also used throughout the disclosures:

The terms “Windows®” and “Microsoft Windows®” are utilized hereininterchangeably to designate any and all versions of the MicrosoftWindows operating systems. By example, and not limitation, this includesWindows XP, Windows Server 2003, Windows Server 2008, Windows Server2012 Windows NT, Windows Vista, Windows 7, Windows 8.x, Windows Mobile,Windows RT, and Windows Embedded. The operation and design of MicrosoftWindows is well documented on the web at msdn.microsoft.com.

The terms “Linux” and “UNIX” are utilized herein to designate any andall variants of Linux and UNIX. By example, and not limitation, thisincludes RedHat Linux, Suse Linux, Ubuntu Linux, HPUX (HP UNIX), andSolaris (Sun UNIX). The design and operation of the Linux operatingsystem is well documented on the web at www.kernel.org.

The terms “iOS” and “MacOS” are utilized herein to designate any and allvariants of Apple's iOS devices and Macintosh operating systems. Byexample, and not limitation, this includes operating systems foriPhones, iPads, iPods, and the Macintosh product lines.

The terms “Android”, “Android OS” and “Android Operating System” areutilized herein to designate any and all variant of Android on any typeof device, whether a mobile phone, phone, tablet, desktop, cameras,embedded device, or other computing or computer device.

The term “node” and “host” are utilized herein interchangeably todesignate one or more processors running a single instance of anoperating system. A virtual machine, such as a VMware, KVM, VirtualBox®, or XEN VM instance, is also considered a “node”. Using VMtechnology, it is possible to have multiple nodes on one physicalserver.

The terms “application” or as appropriate “multi-process application”are utilized to designate a grouping of one or more processes, whereeach process can consist of one or more threads. Operating systemsgenerally launch an application by creating the application's initialprocess and letting that initial process run/execute. In the followingteachings we often identify the application at launch time with saidinitial process.

As an application is a grouping of one or more processes, an applicationmay thus be comprised of one or more other applications, each of whichin turn is comprised of one of more processes. This hierarchy ofapplication may continue to any depth without loss of generality.

In the following we use commonly known terms including but not limitedto “client”, “server”, “API”, “Java”, “process”, “process ID (PID)”,“thread”, “thread ID (TID)”, “thread local storage (TLS)”, “instructionpointer”, “stack”, “kernel”, “kernel module”, “loadable kernel module”,“heap”, “stack”, “files”, “disk”, “CPU”, “CPU registers”, “storage”,“memory”, “memory segments”, “address space”, “semaphore”, “loader”,“system loader”, “system path”, “sockets”, “TCP/IP”, “http”, “ftp”,“Inter-process communication (IPC),” Asynchronous Procedure Calls (APC),“POSIX”, “certificate”, “certificate authority”, “Secure Socket Layer”,“SSL”, MD-5″, “MD-6”, “Message Digest”, “SHA”, “Secure Hash Algorithm”,“NSA”, “NIST”, “private key”, “public key”, “key pair”, and “hashcollision”, and “signal”. These terms are well known in the art and thuswill not be described in detail herein.

The term “transport” is utilized to designate the connection, mechanismand/or protocols used for communicating across the distributedapplication. Examples of transport include TCP/IP, UDP, Message PassingInterface (MPI), Myrinet, Fibre Channel, ATM, shared memory, DMA, RDMA,system buses, and custom backplanes. In the following, the term“transport driver” is utilized to designate the implementation of thetransport. By way of example, the transport driver for TCP/IP would bethe local TCP/IP stack running on the host.

The term “interception” is used to designate the mechanism by which anapplication re-directs a system call or library call to a newimplementation. On Linux and other UNIX variants interception may beachieved by a combination of LD PRELOAD, wrapper functions, identicallynamed functions resolved earlier in the load process, and changes to thekernel sys_call_table. On Windows, interception may be achieved bymodifying a process' Import Address Table and creating Trampolinefunctions, as documented by “Detours: Binary Interception of Win32Functions” by Galen Hunt and Doug Brubacher, Microsoft Research July1999”. Throughout the rest of this document the terminology interceptionto designate the functionality across all operating systems. Theterminology pre-loading is used to designate the process of loading theinterceptors into the application's address space on all operatingsystems.

The term “transparent” is used herein to designate that no modificationto the application is required. In other words, the present inventionworks directly on the application binary without needing any applicationcustomization, source code modifications, recompilation, re-linking,special installation, custom agents, operating system changes, or otherextensions.

The term “fork( )” is used to designate the operating system mechanismused to create a new running process. On Linux, Solaris, and other UNIXvariants, a family of fork( ) calls is provided. On Windows, one of theequivalent calls is “CreateProcess( )”. Throughout the rest of thisdocument we use the term “fork” to designate the functionality acrossall operating systems, not just on Linux/Unix. In general fork( ) makesa copy of the process making the fork( ) call. This means that the newlycreated process has a copy of the entire address space, including allvariables, I/O etc of the parent process.

The term “exec( )” is used to designate the operating system mechanismused to overlay a new image on top of an already existing process. OnLinux, Solaris, and other UNIX a family of exec( ) calls is provided. OnWindows, the equivalent functionality is provided by e.g.“CreateProcessO” via parameters. Throughout the rest of this document weuse the term “exec” to designate the functionality across all operatingsystems, not just Linux/Unix. In general, exec( ) overwrites the entireaddress space of the process calling exec( ). A new process is notcreated and data, heap and stacks of the calling process are replaced bythose of the new process. A few elements are preserved, including butnot limited to process-ID, UID, open file descriptors and user-limits.

The terms “barrier” and “barrier synchronization” are used herein todesignate a type of synchronization method. A barrier for a group ofprocesses and threads is a point in the execution where all threads andprocesses must stop before being allowed to proceed. Barriers aretypically implemented using semaphores, mutexes, locks, event objects,or other equivalent system functionality. Barriers are well known in theart and will not be described further here.

Modern operating systems such as Windows and Linux separate the addressspace into kernel space and user space. Kernel space is the addressspace reserved for running the kernel, kernel extensions, and dependingon operating system, device drivers. User space is the address space inwhich user processes (i.e. applications) run.

To avoid simultaneous use of shared resources in multi-threadedmulti-process applications locking is used. Several techniques andsoftware constructs exists to arbitrate access to resources. Examplesinclude, but are not limited to, mutexes, semaphores, futexes, criticalsections and monitors. All serve similar purposes and often vary littlefrom one implementation and operating system to another. In thefollowing, the term “Lock” is used to designate any and all such lockingmechanism. Properly written multi-process and multi-threaded applicationuse locking to arbitrate access to shared resources

The context of the present invention is an application on the primaryserver (primary application or the primary) and zero, one or more backupservers (also called the backups). While any number of backup-servers issupported the disclosures generally describe the scenario with onebackup. As is obvious to anyone skilled in the art this is done withoutloss of generality.

As part of loading an application for checkpointing the checkpointerkernel module is loaded if not already loaded. The checkpointer kernelmodule provides checkpointing services in coordination with andintegrated with the operating system kernel and user-space interceptors.The checkpoints may be used to start the application again on theprimary, migrate the application to a backup or recover from a fault.The backup and fault recovery use checkpoints taken by the checkpointerto restore the application to a prior state.

Another aspect of the present invention is a resource virtualizationlayer running as part of the interceptors.

A key element of the present invention is thus the use of kernel moduleto capture the state of a running application and save said applicationstate to storage.

Another key aspect of the present invention is to use said storedapplication state to rebuild, also called restore, a copy of theapplication and let the application resume execution from said storedapplication state.

Another aspect of the present invention is that the checkpointingservice is provided as a loadable kernel module, thus providing thecheckpointing service to application as an extension of the operatingsystem kernel without requiring any application customizations orcustomization of system libraries. Yet another aspect of the presentinvention is that the checkpointing service may be built into thekernel, thus also offering the checkpointing service to applications asa built-in operating system service.

A related aspect is that the present invention provides hybridkernel-space and user-space application checkpointing, as the kernelmodule operates along with the user-space library to perform thecheckpointing.

Yet another aspect of the present invention is during a restore from acheckpoint the checkpointer kernel module adjusts kernel state for theparticular application based on the checkpoint.

Another aspect of the present invention is that the checkpointer kernelmodule can be unloaded to free up memory when the checkpointing servicesno longer are needed.

Yet another aspect of the present invention is to optimize checkpointingusing virtual memory area (VMA) information and the associated pagetable entries (PTE), and to use said VMA/PTE to add incrementalcheckpointing.

Another aspect of the present invention is that the_PAGE_DIRTY bit isrepurposed to track dirty pages for checkpointing.

A further aspect of the present invention is that it can be provided oncommodity operating systems such as Linux and Windows, and on commodityhardware such as Intel®, AMD® (Advance Micro Devices), SPARC® (ScalableProcessor Architecture), ARM® (Advance Risk Machines), and MIPS®(Microprocessor without Interlocked Pipeline Stages). The presentinvention thus works on commodity operating systems, commodity hardwarewith standard (off the shelf) software without needing any furthermodifications.

Further aspects of the invention will be brought out in the followingportions of the specification, wherein the detailed description is forthe purpose of fully disclosing preferred embodiments of the inventionwithout placing limitations thereon.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

The invention will be more fully understood by reference to thefollowing drawings which are for illustrative purposes only:

FIG. 1 is a block diagram of the core system architecture

FIG. 2 is a block diagram illustrating a pair of primary and backup

FIG. 3 is a block diagram illustrating virtual and physical memory andtheir mappings

FIG. 4 is a block diagram illustrating use of the virtual file system

FIG. 5 is a block diagram illustrating the checkpointing process

FIG. 6 is a block diagram illustrating the interaction of the barrier,checkpointer threads, and kernel module.

FIG. 7 is a block diagram illustrating the barrier.

FIG. 8 is a block diagram illustrating the structure of a checkpoint.

FIG. 9 is a block diagram illustrating restoring a checkpoint.

FIG. 10 is a block diagram illustrating various deployment scenarios.

FIG. 11 is a block diagram illustrating breakdown of virtual addresses

FIG. 12 is a block diagram illustrating VMAs

FIG. 13 is a block diagram illustrating checkpointing by VMA

FIG. 14 is a block diagram illustrating incremental checkpointing

FIG. 15 is a block diagram illustrating restoring from a checkpoint

DETAILED DESCRIPTION OF THE INVENTION

Referring more specifically to the drawings, for illustrative purposesthe present invention will be disclosed in relation to FIG. 1 throughFIG. 15 It will be appreciated that the system and apparatus of theinvention may vary as to configuration and as to details of theconstituent components, and that the method may vary as to the specificsteps and sequence, without departing from the basic concepts asdisclosed herein.

0. Introduction

The context in which this invention is disclosed is an applicationrunning on a primary server. Without affecting the general case ofmultiple primary applications, the following disclosures often depictand describe just one primary application. Multiple primary applicationsare handled in a similar manner.

Likewise, the disclosures generally describe applications with one ortwo processes; any number of processes is handled in a similar manner.Finally, the disclosures generally describe one or two threads perprocess; any number of threads is handled in a similar manner

1. Overview

FIG. 1 illustrates by way of example embodiment 10 the overall structureof the present invention. The following brief overview illustrates thehigh-level relationship between the various components; further detailson the inner workings and interdependencies are provided in thefollowing sections. FIG. 1 Illustrates by way of example embodiment 10 aprimary server 12 with an application 16 loaded into system memory 14.The application 16 is comprised of two processes; process A 18 andprocess B 20. Each of the two processes has two running threads. ProcessA contains thread T0 22 and thread T1 24, while process B is containsthread T3 26 and thread T4 28. As part of loading the application 16 aninterception layer (IL) 23, 29 is pre-loaded into the address space ofeach process. Interception Layer 23 for process A is preloaded intoprocess As address space and Interception Layer 29 is preloaded intoprocess B's address space. The system libraries 36 are generallyinterposed between the application 16 and operating system 38. Alongsidewith the operating system 38 are the loadable kernel modules 30.

System resources, such as CPUs 46, I/O devices 44, Network interfaces 42and storage 40 are accessed using the operating system 38 and theloadable kernel modules 30. Devices accessing remote resources use someform of transport network 48. By way of example, system networking 42may use TCP/IP over Ethernet transport, Storage 40 may use Fibre Channelor Ethernet transport, and I/O 44 may use USB.

The architecture for the backup is identical to the primary,

In the preferred embodiment storage 40 is external and accessible byboth primary and backups over a network 48.

FIG. 1 illustrates the system libraries 36 as separate from theapplication 16 and the individual processes process A 18 and process B20. The system libraries are generally shared libraries. For clarity ofpresentation, the system libraries are depicted outside the addressspace of the individual processes, even though some library state andcode is present within the address space of the individual applicationprocesses.

In an alternate embodiment the functionality of the kernel module 30 isbuilt into, i.e. compiled into, the kernel. This eliminates the need toload the kernel module, at the expense of being a custom kernel. Thepreferred embodiment disclosed herein provides checkpointing services asa kernel module, but it is obvious to anyone with ordinary skills in theart the kernel module functionality could be compiled into the kernel asdisclosed for the alternate embodiment.

FIG. 2 illustrates by way of example embodiment 60 a primary server 62and its corresponding backup server 82 working as a pair of primary andbackup. The primary application 64 is comprised of two processes;process A 66 and process B 68, each with two running threads and aninterception layer. System libraries 70 on the primary are interposedbetween the application 64 and the operating system kernel 76. Loadablekernel modules 72 are loaded by the kernel 76 as needed.

Using a similar architecture, the backup server 82 contains the backupapplication 84 comprised of process A 86 and process B 88 each with twothreads and interception layer. System libraries 90 are interposedbetween the application 84 and the operating system kernel 96. Loadablekernel modules 92 as loaded by the kernel 96 as needed.

Primary and backup communicate over one or more network connections 98.

2. Kernel Structure

Linux and Windows are two dominant operating systems with differentkernel architectures. Linux is build around a monolithic kernel withloadable kernel modules, while Windows is built around a micro kernelwith a core executive. As with Linux, Windows supports loadable kernelmodules and kernel-mode drivers. Both Windows and Linux thus enabledrivers and modules to run in kernel space.

2.1 Memory

On the Linux operating system applications running in user spacecommunicate with the kernel through defined kernel interfaces. Thekernel interfaces are generally concealed by the system libraries as faras the applications in user space are concerned. On Windows, the Windowssystem libraries exemplified by the Win32 and Posix subsystems likewisemanage the interfaces to kernel functionality. Generally, a user-spaceapplication doesn't need to concern itself with kernel behavior; itsimply requests kernel services through the system libraries.

Both the Linux and Windows kernels and system libraries are welldocumented and only the elements relevant for the present invention willbe further described.

On a 32-bit operating system an application can generally address thefull 4 GB of memory corresponding to the 32 bits of address space, evenif the underlying hardware provides less than 4 GB of memory. As apractical matter Linux reserves the upper 1 GB of memory for theoperating system, while Windows generally reserves the upper 1 or 2 GBfor the operating system. The available physical memory is first mappedinto the address space of the kernel, and the kernel is then proving aflat virtual 4 GB address space for the application. For 64-bitoperating systems and hardware similar considerations apply, other thanthe address space is 64 bits instead of 32 bit. The kernel maintainssaid mapping between application virtual addresses and physical memoryaddresses. The virtual address space of is broken into equal sizedportions called pages by the kernel. Physical memory is likewise brokeninto similar sized pages, called page frames. On IA32 systems the pagesize is 4 kb (4096 bytes). Other architectures may have different pagesizes. In the following PAGE_SIZE is used to designate the page size onall platforms. The present invention relies on this mapping to locateand extract application memory image as part of checkpointing whilerunning within the context of the kernel. Generally, the address spacewhen viewed from an application process's perspective is called processvirtual memory or process address space, while the underlying physicalmemory of the host system is called physical memory.

FIG. 3 illustrates by way of example embodiment 100, the relationshipbetween process virtual memory and physical memory. Process A's virtualmemory 102 is broken into pages. Each allocated page in virtual memoryis mapped to a page in physical memory 104. By way of example page zeroin process A's virtual memory 102 is mapped to page frame 1 in physicalmemory, and page 3 in process A's virtual memory is mapped to page frame3 in physical memory. Process B's virtual memory 106 page address zerois mapped to page frame 1 in physical memory while page 2 in virtualmemory is mapped to page frame 7 in physical memory. Page frames inphysical memory may be shared between two processes as illustrated forpage frame 1. While nomenclature and some of the finer details varybetween operating systems, the disclosures above generally describe themechanics on Linux and Windows and the nomenclature set forth is used inthe following for teachings on both Linux and Windows.

2.2 Resources and Processes

Each process has resources attached. By way of example, on the Linuxoperating system, information related to a running process is capturedand stored in the task_struct structure or can be retrieved via saidtask_struct. Some of the data kept in the task_struct is processexecution status, process hierarchy, allocated memory, processcredentials, memberships, signals, resources in use and threadinformation. As the task_struct is a kernel data structure theinformation stored within is generally not accessible by applicationsrunning in user space. Applications can access some of the elementsusing system calls abstracted out within the system libraries. Datawithin the task_struct, however, is accessible when running in kernelmode, a fact the present invention builds on.

The kernel also maintains a list of all running processes called thekernel task list. In newer versions of the Linux kernel the kernel tasklist is private and can therefore not be directly accessed. The presentinvention does therefore not rely on the global kernel task list; ratherit relies only on the local task_struct for the process currently beingaccessed and uses a user-space library to coordinate access to differentprocesses.

2.3 Device Drivers

Device drivers provide implementation of hardware specific interfaces.Both Linux and Windows provide generic device support and impose certainarchitectural constraints on drivers in order to provide universalsupport for a large number of drivers. By way of example, operatingsystems provide device drivers for access to hard disks, CD ROM drives,USB, internal and external peripherals. By way of example, when anapplication opens a file, a device driver is ultimately responsible forretrieving or writing data to said file.

Device drivers thus enable access to very specific resources usingwell-defined mechanisms offered by the kernel and abstracted out by thesystem libraries. Device drivers run as part of the kernel eitherbuilt-in or dynamically loaded as a loadable kernel module. As such,device drivers have access to the kernel internals and can inspect andmanipulate kernel structures. The present invention builds on thesefacts to offer checkpointing services, where key elements of saidcheckpointing services are provided as a checkpointing device driver.

FIG. 4 illustrates by way of example embodiment 120 the structure on theLinux operating system. An application 124 runs in user space 122.Kernel functionality is accessed either directly from the applicationthrough sys calls, or through the system libraries 126. Within kernelspace 128, the System Call Interface 130 receives and processes thesystem call. The system call is translated into a call into the kernel132, where it's processed, eventually passed through one or more devicedrivers 134 and the underlying hardware 144 is accessed. For manydevices, the Virtual File System (VFS) is used a common abstraction. TheVFS 138 is a kernel module 136 and provides a “standard file system”that defines services and interface many devices must support. VFS isgenerally mounted over a file systems such as ext3, ext4, XFS, andReiser 140 for block devices, and directly over the character driver fora character device. Even virtual file system, such as the /proc filesystem, is provided within the context of VFS. VFS and the extended filesystems ultimately rely on one or more device drivers 142 to access thestorage media 144.

As illustrated in the example embodiment 120 on FIG. 4 user-spaceapplication never directly access the device driver; rather the devicedriver is accessed going through standard kernel interfaces (System CallInterface) using abstractions such as VFS. The present inventionutilizes the VFS abstraction as well, and provides checkpointing thatfrom the applications perspective looks like reading files from a filesystem. Likewise, restoring checkpoints looks like writing a file to afile system.

3 Checkpointing Device Driver Introduction

In the following disclosures we follow the Linux naming convention,where device drivers are installed in the “/dev” directory. In thefollowing disclosure the checkpointer device driver is named “ckpt” andthe full pathname is thus “/dev/ckpt”. The actual device name is notimportant as long as it's different from other device names supported bythe kernel and available in the /dev or other directory.

By way of example, /dev/ckpt is implemented as a character device. Thedisclosed functionality of /dev/ckpt could also be implemented as ablock device, which is obvious to anyone with ordinary skills in theart. Therefore, the preferred embodiment uses a character device whilean alternate implementation uses a block device. Character devices andblock devices are both known in the art and the following disclosureswill thus focus on aspects relevant to the present invention.

A character device is identified by its major device number. By way ofexample, the following teachings use a major device number ofCKPT_MAJOR_NUM. For testing purposes the locally reserved numbers in therange 240 to 254 can be used. By way of example on the Linux operatingsystem, the device node is create as

# mknod/dev/ckpt c CKPT_MAJOR_NUM 0

3.1 File Operations

The internal structure of a device driver must declare thefile_operations supported by the device. By way of example, ckptfile_operations is defined as follows:

struct file_operation ckpt_fops={

.owner=THIS_MODULE,

.open=ckpt_open,

.release=ckpt_release,

.read=ckpt_read,

.write=ckpt_write,

.ioctl=ckpt_ioctl,

.seek=ckpt_seek,

};

Here each of the device driver functions ckpt_open, ckpt_release,ckpt_read, ckpt_write, ckpt_ioctl and ckpt_seek are assigned to theirrespective file_operations.

ckpt_open is used to initialize the device driver, ckpt_release torelease all resources associated with the device before taking thedevice down. ckpt_read and ckpt_write are used to read and write bytesof data respectively, ckpt_ioctl is used to query the device driver forspecial status information, and ckpt_seek is used to reset the currentposition in the device. Finally THIS_MODULE identifies the module.

In addition to open/read/write/release calls, the /dev/ckpt alsocontains kernel module initialization. The module initialization andtermination is often called module init( ) and module exit( ) inreference to the commonly found declarations in kernel modules. By wayof example the following teachings use the terminology ckpt module init() and ckpt module exit( ) or said functions of within the context of thepresent invention.

In the preferred embodiment ckpt module init( ) call registers thedevice with the operating system.

-   static int_init ckpt_module_init(void)-   {    -   int ret;    -   if ((ret=register_chrdev(CKPT _MAJOR_NUM, “ckpt”,        -   &ckpt_fops))<0)            -   printk(KERN_ERR “ckpt_module _init: % d\n”,ret);    -   return ret;-   }

The register_chrdev call provides the registration information thatallows the operating system to map the /dev/ckpt to the actual devicedriver.

Similarly ckpt_module_exit( ) unregisters the module with a preferredembodiment as

-   -   static void_exit ckpt_module _exit(void)    -   {        -   unregister_chrdev(CKPT _MAJOR_NUM, “ckpt”);    -   }

3.2 Conceptual Use of /dev/ckpt

By way of example, the /dev/ckpt checkpointer module may be loaded usingthe standard Linux command insmod and the /dev/ckpt checkpointer may beunloaded using the Linux command rmmod. Both command are well known inthe art and will not be described further herein.

The /dev/ckpt checkpointer is accessed from user space like any otherfile or device. In the preferred embodiment the checkpointer is opened,ignoring error handling, using code similar to

-   -   pCkpt=open(“/dev/ckpt”,O_RDWR);    -   and the checkpointer is closed using code like    -   close(pCkpt);

In the preferred embodiment, a checkpoint is created by readingfrom/dev/ckpt using code like

-   -   readCount=read(pCkpt,buffer,size);    -   and a restore from checkpoint is accomplished by writing        to/dev/ckpt using code like    -   writtenCount=write(pCkpt,buffer,size);

FIG. 5 illustrates by way of example embodiment 160 the overall flow ofthe checkpointing process. First 162 the /dev/ckpt checkpointer moduleis loaded using, by way of example, the insmod command. The applicationis then started 164. As part of loading the application interceptors areloaded containing the user-space components of the checkpointer. Thecheckpointer is started 166 by way of the /dev/ckpt open command. Theapplication is then let run 168 for some period of time. If theapplication is not finished processing 170 the checkpointer is activated172. A checkpoint is taken through the /dev/ckpt read command 172, andthe application let run again 168. If the application is done running170 the checkpointer is closed 174, the application exited 176 and the/dev/ckpt checkpointer unloaded using by way of example rmmod.

The finer details of the checkpointing process and its interaction withthe user-space interceptors are disclosed in the next sections.

4.0 User-Space Shared Library

On the Linux operating systems new processes are created by the combineduse of fork( ) and exec( ). On Windows, processes may be created byusing the CreateProcess( ) call. As described above, the disclosures ofthe present invention use the terminology fork( ) and exec( ) todesignate the process creating interfaces on all operating system.

The user-space functionality is pre-loaded as a library as part ofinitially loading the initial application process and later every time anew process is created. The user-space library provides the followingfunctionality

-   -   a. Creation of a dedicated checkpointing thread for each process    -   b. Preparation of Barrier for process and threads within process    -   c. Collection of checkpoint data by calling/dev/ckpt    -   d. Flushing of system resources    -   e. Resource virtualization, Application Virtualization Space        (AVS).

As part of checkpointing, the present invention deterministically haltsexecution of the application being checkpointed. Halting of amulti-process multi-threaded application is achieved using barriersynchronization (as defined in “Brief summary of Invention”). The use ofBarrier synchronization for halting a multi-process multi-threadedapplication is taught in U.S. Pat. No. 7,293,200 Neary et al andincluded in its entirety by reference. The use of a barrier within thecontext of the present invention is disclosed below.

Where Neary et al. provides the actual checkpointing facility in a“check point library” running in user-space, the present inventionprovides the core checkpointing facility as a kernel module. The presentinvention utilizes the barrier from the Neary, and eliminates the needfor the checkpointer to run in user space. Moving the checkpointer fromuser-space to kernel-space has several advantages: 1) it essentiallyremoves the dependency on the system libraries found in Neary, 2) iteliminates the need to closely track application and libraryinteractions, as all checkpointing is done in the kernel sitting underthe system libraries, and 3) it automatically includes all memoryallocated in the address space of the application without the need tospecial case stacks, code, text and shared memory.

The present invention creates a checkpointing thread per process astaught in Neary. However, contrary to Neary, the checkpointer threadcalls the /dev/ckpt to collect the checkpoint and does not itself createthe checkpoint for its parent process.

FIG. 6 illustrates by way of example embodiment 180 an application 182with two processes, process A 184 with two threads T1 and T2 and processB 186 with two threads T3 and T4. Process A additionally contains thecheckpointer thread C-A 191 and process B contains checkpointer threadC-B 193. The Barrier 188 indicates a place in the execution of processA, process B and threads T1, T2, T3 and T4 where all processes andthreads are halted. With all activity in user space halted at thebarrier, the kernel module checkpointer (/dev/ckpt) 198 can collect allrelevant process state and build a checkpoint. Each process collects itsown checkpoint via the kernel module as indicated for process A 190 viaits checkpointer thread C-A 191 and process B 192 via its checkpointerthread 193

In contrast to Neary and Backensto, the user-space library of thepresent invention contains only the barrier, checkpointer thread,flushing, AVS and collection of checkpoint data. There is no need tokeep track of memory allocations, including those hidden within thesystem libraries, as all checkpointing data is collected in the kernel.This is a dramatic simplification over the user-space checkpointer anddoes not require customized system libraries.

4.1 Flushing of TCP/IP

If, by way of example, an application uses TCP/IP with thesocket-programming model, select TCP and socket state and data areimplicitly included in the application state, even though the data mayreside in the TCP stack or associated device driver. As previouslydisclosed, the present invention does not attempt to access kernelstructures for non-application processes and thus approaches the problemof the TCP and socket state from the application layer. Whileapplication and socket buffers generally are included in theapplications memory image that is not the case for kernel and devicebuffers related to the lower levels of the TCP/IP stack and the networkdevice itself. In the preferred embodiment the approach taken is toflush all TCP and socket communication in order to move data through thebuffers thereby ensuring that the buffers outside the application are ina consistent state with respect to the application. Flushing of TCPsockets may be achieved in a number of ways including but not limited toclosing the socket, which forces a flush and then reopening the socket,setting the TCP NODELAY socket option or through application specificflush operations. In an alternate embodiment the TCP and device kernelbuffers are extracted in kernel module without requiring any flushingoperations.

Resources accessible from the task_struct can be flushed directly by thekernel module checkpointer and does therefore not need flushing fromuser space. Only resources with kernel functionality outside thetask_struct, such as TCP/IP, require flushing from user space.

Generally, resources are flushed as part of the checkpointing process inorder to move data out of low-level buffers located outside the addressspace of application processes, through the operating system and onto orinto the devices.

4.2 Application Virtualization Space—AVS

Instantiated resources generally are only meaningful in the context ofthe operating system where the application is running. By way of examplethe operating system assigns each process a unique process ID (PID). Atany point in time each process on a particular node has a unique processID. If the process is stopped and re-started it may not get the samePID. Likewise, if the process is started on a different node, the PIDmay be different. Some applications query the operating for their ownPID and use the PID for a variety of purposes. For a checkpoint image tobe meaningful, it's thus important that the PID is presented toapplications with the value it had at the time of taking the checkpoint.

A structure called the Application Virtualization Space (AVS) is used,to continue to example, to store the PID at the time the process iscreated and when the PID is accessed at later time through the definedAPIs, to retrieve it from the AVS. At the time the process, and thus thePID, is first created the PID is recorded within the AVS for theparticular process and kept associated with process as long as it'srunning. When the process calls getpid( ) to get the associated PID, thevalues stored in the AVS is returned, as opposed to the PID provided bythe underlying operating system. This ensures that dependencies on PIDsare virtualized and preserved across checkpoints. Similar teachingsapply to thread IDs (TIDs), and other system constructs such as IPC,signal, and named semaphores.

Files are handled in a similar manner. By way of example, the file name,path, and attributes are stored in the AVS. As part of restoring anapplication from a checkpoint, the contents within the AVS is used torecreate the data structures used by the operating system to representthe file, i.e. the file's name, path, and attributes. Current filestate, such as file position, is contained within the checkpoint anddoes not need to be included in the AVS; the AVS only needs the minimalinformation required to re-create the resource and the checkpoint thencontains the current state.

Similarly, network state such as state of sockets, may be included inthe AVS. As with files, only the data relevant to re-creation of theresource is required, while current state is contained within thecheckpoint.

The AVS entries are created when resources are created or opened, andremoved from the AVS when the resource is closed. By way of example,this means that the AVS entry for a file is created when the file isopened, and removed when the file is closed. The AVS is generally notinvolved with individual file operations, such as reading or writingdata, only in creating file pointers that are meaningful in the contextof the operating system and the associated checkpoint.

The AVS thus contains an application-specific private name space forresources, and maintains mappings between resources within the initialinstances of application resources and their possibly different valueswhen running in a different environment at another time. In the exampleembodiment, the AVS mappings are implemented using a hash table. Inother embodiments the AVS is implemented using a database or customcoding.

5. Checkpointing in Kernel Module/Dev/Ckpt

As disclosed in section 3 the read( ) method of /dev/ckpt is called toretrieve checkpoint data from the kernel. As disclosed in section 4,read( ) is called within the barrier, so the underlying process is notexecuting application code.

As taught in section 2 all state information for a particular process isstored in or can be found from the process's task_struct structure. Byway of example on the Linux operating system, the task_struct for thecurrent process can be found by the current macro. In continuation ofsection 3 an example embodiment, without error handling, of the readfunction for /dev/ckpt is

-   -   static ssize_t ckpt_read(struct file*filp,        -   char*ubuff, size_t count, loff_t*f_pos)    -   {        -   // pMem is a pointer to memory seen from        -   // the kernel        -   Set pMem to the current address in flip        -   // copy to user space        -   eCount=copy_to _user(ubuff,pMem, count)        -   nBytes=(eCount==0)? count: (count+eCount);        -   f_pos+=nBytes;        -   return count    -   }    -   copy_to _user( ) returns zero if count bytes have been copied or        -eCount if only ecount bytes were transmitted. nBytes represents        the number of bytes actually transferred to user space. In an        alternate embodiment a page is memory mapped into user-space and        the checkpoint data copied directly into said memory mapped        page.

Generally, attempting to access a memory location not allocated to theprocess causes a page fault. However, copy_to _user( ) specificallydeals with this issue by validating addresses before the copy operation.If the page is invalid, i.e. the linux function access_ok( ) returnszero, copy_to _user( ) does not attempt to access the page, and thefunction returns without attempting to copy data. It is thus notnecessary to check every page before attempting to copy.

In an alternate preferred embodiment, the page data is transferred touser-space using memory mapped pages.

Each of these steps are now disclosed in further detail.

5.1 Collecting Memory State

The process's task_struct contains a data structure mm_struct whichcontains the relevant information regarding said process's virtualmemory. The mm_struct contains the kernel data structures necessary todetermine which memory pages are used by the process.

By way of example, the start and end of code may be found using thestart code and end code variables, the start and end of data with thestart data and end data, the start and end of the heap with start brkand brk. Similar variables may exist for stack, environment andarguments.

As memory is organized in pages within the kernel, the most efficientoperation of /dev/ckpt is in units of memory pages, as opposed to unitsof bytes. However, since the core device read( ) operations operates inunits of bytes, the calling process preferably aligns calls to read( )on page boundaries and requests blocks of bytes with the size of a page.In the preferred embodiment, a call to read( ) may thus look likeread(filp,ubuff,PAGE_SIZE,p_pos). The read( ) function return the numberof bytes read as a means of communicating success or error. If e.g. zero(0) is returned, it means that zero bytes were actually read into ubuff.

By way of example embodiment, and in continuation of the example insection 3.2, the calling process may call /dev/ckpt with pseudo codelike

-   -   unsigned byte ubuff[PAGE_SIZE];    -   //open device    -   //set/reset filepos to zero.    -   seek(fp,0);    -   for(int i=0; i<maxMem; i+=PAGE_SIZE)    -   {        -   count=read(fp,ubuff,PAGE_SIZE)        -   if (count>0) save checkpoint of page            -   stored in ubuff    -   }

The /dev/ckpt driver may optimize the checkpointing process by using thepreviously disclosed information about start and end of code, text,stack etc. The checkpointer may thus check the address of the currentread request to verify that it falls outside the pages used by theprocess and if that is the case, immediately return a value of zero toindicate that no application process data was available for therequested read( ) operation. A related optimization is that on someoperating systems such as Linux a certain number of pages in low memoryare left unused to catch null-pointers. Similarly, the operating systemkernel may, by way of example, reside in the upper or lower 1 GB of thememory address space and may thus be skipped by the checkpointer. Eachoperating system is slightly different, and may offer opportunity forsuch simple optimizations of the checkpointing process.

In the example, the user-space buffer ubuff is fixed and pre-allocated,and thus automatically included in the checkpoint as well. However,since ubuff is used in a transient manner with data flowing through, itscontent does not need to be preserved as part of a checkpoint.

5.2 Flushing of System Resources

In the preferred embodiment disclosed above, TCP/IP connections areflushed from the user-space library. Other system resources, such as IPCis generally contained within the task_struct and can thus be capturedby /dev/ckpt by accessing resource within the address space of thecurrent process. Examples include, but are not limited to the System Vsemaphores sysvmem, open files identified by *files and state of signalhandlers. In these cases /dev/ckpt can access the resource internals andflush buffers as supported by the device.

5.3 Integration with User-Space Library

As disclosed above the kernel module checkpointer /dev/ckpt is called bythe checkpointing thread for a process to collects a specified number ofmemory bytes, typically a page, for said process. The /dev/ckptcheckpointer does thus not need to access the address space of any otherprocesses, nor does it need to know about other processes.

Conversely, the user-space library is responsible for coordinationacross all processes comprising an application. The barrier containedwithin the user-space library brings every application process andthread into a stable locked state where the application processes andthreads do not execute. With the application essentially halted, thecheckpointer thread for each process can call /dev/ckpt and assemble thememory pages comprising the application image.

5.4 Deterministic Halting at the Barrier

The inner working of the barrier as it relates to the checkpointingprocess is now disclosed. FIG. 7 illustrates by way of exampleembodiment 200 an application with one process 202 containing to processthreads T0 206 and T1 208 and the checkpointer thread 204 as disclosedabove. Multiple processes are handled in a similar manner.

Initially the application the process is running A 210 with thecheckpointer thread 204 and the two process threads 206, 208 operating.At some point a checkpoint is triggered 224. The checkpoint triggeractivates the barrier and transitions into waiting B 212 for T0 and T1to stop at the barrier. When both T0 and T1 are stopped at the barrierthe checkpointing thread 204 starts assembling the checkpoint C 214. Asdisclosed previously, the checkpoint assembly comprises collectingmemory and kernel state for the process 202. Kernel buffers are flushedand relevant state information saved along with the memory state. Saidcheckpoint state is then written to disk D 216. In the preferredembodiment disclosed previously states C 214 and D 216 are interleaved,in that one or more memory pages repeatedly are collected then writtento disk. That loop is indicated with the callout 226 next to state C andD. If a kernel device had to be halted as part of flushing, the deviceis activated again E 218, and the checkpointer transitions to state F220 where individual threads T0 and T1 are released in reverse orderfrom the barrier. Upon being released, the application process 202 runsagain G 222.

5.5 Triggering of Checkpoints

Checkpoints may be triggered in a variety of ways. By way of example, acheckpoint may be triggered after a certain elapsed time, if the CPUtemperature exceeds a certain level, if a memory threshold is met, if astorage threshold is met, if a network traffic threshold is meet, if anew server as added, after changes in configuration, in response to anexternal script or program, or manually by the administrator. Theteachings of the present invention are independent of how thecheckpoints are triggered and triggering will therefore not be furtherdescribed.

6. Management of Memory Using Page Tables and PTEs

As taught above, memory is divided in the pages and presented to anindividual process as a virtual address space.

Page tables are used to establish a mapping between a user-spaceprocess's virtual address space and the physical memory of the system.The use of page tables is well known in the art(http://en.wikipedia.org/wiki/Page_table) and will only be described tothe extent that is required to fully disclose the present invention.Page tables are often multi-level; by way of example Linux generallyuses four levels to fully break down an address across page tablelevels. Modern CPUs, such as x32/x64/AMD and SPARC, have memorymanagement units (MMUs) that traverse the page tables to make memoryaddressing efficient and fast.

The levels of the page table hierarchy are often given names. FIG. 11illustrates by way of example embodiment 320 the names commonly used onLinux. The basic size of an address 322 is given as ‘architecture size’.It's generally 32 or 64. PGD 324 is the Page Global Directory; PUD 326is the Page Upper Directory; PMD 328 the Page Middle Directory and PTE330 the Page Table Entry. Offset 332 designates the offset within thePTE. The page table designated by PTE contains the actual informationabout the page including access flag. By way of example, the PTEcontains the following flags relevant for the teachings of the presentinvention:

-   -   _PAGE_ACCESSED which generally is set automatically by the CPU        to designate that the page has been accesses with a read or        write operation    -   _PAGE_DIRTY which generally is set automatically by the CPU to        indicate that the page “dirty”, i.e. that the page's content has        been modified.        By way of example, the Linux operating system provides a family        of functions to access and work with said PTE page table. Some        of the functions relevant for the present teachings are    -   pte_present( ): is the page present in the system    -   pte_read( ): can page be read    -   pte_write( ): can page be written    -   pte_dirty( ): is page dirty        The entire family of pte calls is documented in the Linux kernel        sources at www.kernel.org.

As a process executes, the CPU continuously updates the _PAGE_ACCESSEDand _PAGE_DIRTY bits. Most modern CPUs provide _PAGE_ACCESSED and_PAGE_DIRTY flags, or they are implemented in software by the operatingsystem. In the following _PAGE_ACCESSED and _PAGE_DIRTY is used todesignate said functionality on all CPUs independently of whether theCPU provides native support or the support is provided by the underlyingoperating system. These teachings are combined with section 9 to furtheroptimize the checkpointing process.

To find the page corresponding to a virtual memory address, the pagetable needs to be traversed. An example embodiment of Traversing thepage table without error handling, is given in the following pseudocode:

-   -   static struct page find_page(struct mm_struct mm,        -   unsigned long address);    -   {        -   pgd=pgd_offset(mm, address)        -   pmd=pmd_offset(pgd,address);        -   pte=pte_offset(pmd,address);        -   return pte_page(pte);    -   }

Where pgd, pmd, and pte designate the Page Gobal Directory, Page MiddleDirectory, and Page Table Entry introduced above. The incrementalcheckpointer collects and uses state from the individual pages. Thepreviously disclosed algorithm is used in a preferred embodiment to findthe page associated with a particular virtual memory address.

7. Structure of Checkpoint

The checkpoint of a multi process application is comprised of stateglobal to the operating system, state shared across all processes withinthe application, and state for each process including the state of allthreads within said process.

FIG. 8 illustrates by way of example embodiment 240 the structure of anapplication checkpoint 242. The checkpoint is comprised of state globalto the application on the operating system 244. This includes the AVSand other data that is system global. The process hierarchy 246 containsthe hierarchy of processes comprising the application and shared data248 contains information on all shared data between processes within theapplication. Examples of shared data include names of shared semaphores,shared memory etc. The checkpoint data for the individual processes 250is comprised of individual checkpoints for each process. By way ofexample, the checkpoint for process 1 252 is followed by the checkpointfor process 2, and so on until the last process, process ‘n’ 256.

FIG. 8 also illustrates by way of example embodiment 240 the layout ofthe checkpoint for individual processes. By way of example thecheckpoint for process 1 252 is comprised of the following data. Firstthe total number of pages in the checkpoint 260 is provided as a meansof boundary checking and error recovery. Each memory page included inthe checkpoint is identified by a page info block 263. A page info block263 contains information describing the page, including but not limitedto page start address, physical page state information as contained inthe PTE, and virtual address space page state information as containedin the associated VMA. Disclosures related to VMA are presented in thenext section.

By way of example, the page info for the first page in the checkpoint262 is followed by the data contained in said first page 264. This isfollowed by the page info for the second page 266 in the checkpoint andthe data contained in said second page 268. The same layout is followeduntil the last page info 270 is reached and the data contained in saidlast page 272.

The layout of the individual process checkpoints in the just disclosedexample embodiment allows for saving of only those pages used by theapplication, and thus supports the checkpointer optimizations previouslydisclosed in section 5.1.

Methods for storing the hash tables as used by AVS are well known in theart and will therefore not be discussed in further detail. Likewise,methods for storing trees, as used by the process hierarchy, are wellknown in the art and will therefore not be discussed further

8. Checkpoint Optimizations Using VMAs

Within the virtual address space of a process Linux uses the techniqueof Virtual Memory Areas (VMAs), also called Regions, to manage memoryand further refine the access rights to individual pages or groups ofpages. In the disclosures the terms Region and VMA are usedinterchangeably to represents an interval of address space that is amultiple of the underlying page size. VMAs do not overlap and all pageswithin a VMA have the same attributes.

By way of example, on Linux the mm_struct data structure contains a listof vm_area_structs (called *mmap) with the elements in the listrepresenting a VMAs and their underlying component VMAs. Thishierarchical layout makes it easy to assemble a checkpoint comprised ofall process pages.

-   -   By way of example, the relevant element of the mm_struct data        structure is    -   struct mm_struct {        -   struct vm_area _struct*mmap;        -   . . .    -   };

While the relevant elements of the vm_area _struct are:

-   -   struct vm_area _struct {        -   struct mm_struct*vm_mm;        -   unsigned long vm_start; //starting address        -   unsigned long vm_end; //end address+1        -   struct vm_area _struct*vm_next; //next element        -   . . .    -   };

FIG. 12 illustrates by way of example embodiment 340 the use of VMAs tooptimize the checkpointing process. The process's full virtual addressspace 341 contains three VMAs; first VMA 342, second VMA 344 and thirdVMA 346. The underlying mm_struct 358 from the process contains the*mmap pointer 357 to the list of vm_area_structs representing theunderlying VMAs comprising the processes address space. First vm_area_struct 352 contains vm_start 348 and vm_end 350 pointers. The *vm_next351 pointer, points to the next vm_area _struct. The secondvm_area_struct 354 and the third vm_area _struct likewise identify thevm_start and vm_end memory locations and provide a pointer to the nextvm_area _struct if applicable.

Based on the disclosures above, checkpointing may thus be optimized bytraversing the list of vm_area_structs found from the mm_struct (*mm)located within the task_struct. Contrary to the teachings in section 5,by traversing the VMAs each set of pages are known to be in use by theprocess and will thus not cause page faults that need masking. This,however, means that the user-space checkpointing disclosed in section5.1 no longer is adequate as the calling read( ) function no longer canpre-determine the address of the page being read( ), i.e. checkpointed.The following disclosures describe a preferred embodiment with of/dev/ckpt and associated user-space calling method to support dynamicand interactive checkpointing optimized by using the VMAs.

By way of example, we examine the example embodiment 340 on FIG. 12 andlook at the first memory VMA 342. By way of example said first memoryVMA is comprised of three pages of memory. First page 360, second page362, and third page 364. The start address of the first VMA 348corresponds to the start of the first page 366, while the end of thefirst memory VMA 350 corresponds to the end of the third and last page368 in the VMA. To read( ), i.e. checkpoint, said first VMA, the read( )call is called three times, one for each page, and the offset (*f_pos)advanced to the next page. For the very first call, the calling functionseeks to the position of zero, indicating that this is a new checkpoint.This in turn signals the checkpointer to identify the first memory VMAand read the first page in said first VMA. Upon completing the read( )operation the offset is advanced to the next page which is either in thesame VMA or the next VMA in the list. When a read( ) reaches the end ofthe list of VMAs and has read the last page, zero is returned toindicated “end of checkpoint”.

An example embodiment in pseudo code is:

-   -   static ssize_t ckpt_read(struct file*filp,        -   char*ubuff, size_t count, loff_t*f_pos)    -   {        -   if (f_pos==0)        -   {//new checkpoint            -   //find start page for first vma            -   pMem=address of first page in first vma        -   }        -   else        -   {            -   pMem=address by f_pos        -   }        -   If (f_pos indicates end of address space)            -   return 0; //we're done        -   //read page and copy to user space        -   eCount=copy_to _user(ubuff,pMem, count)        -   nBytes=(eCount==0)? count:(count+eCount);        -   f_pos+=nBytes;        -   //move to start of next page or end address        -   return count    -   }

FIG. 13 illustrates by way of example embodiment 380 the pseudo code infurther detail. First the ckpt_read( ) is called 382. It is thendetermined if there are any virtual memory pages 383. This may beaccomplished by checking the size of the *mmap list in the mm_struct. Ifthere are no pages to be checkpointed the nBytes variable is set to zeroand the function returns. If there are pages 383 it is then determined386 if this is the start of a new checkpoint, i.e. if this is the firstpage in the checkpoint. The start of a new checkpoint is identified byf_pos being zero. The first page in the first VMA is then identified 388via the mm_struct and the vm_area _struct as disclosed above. If thecurrent call does not refer to the first page in the checkpoint 386, thepage to be checkpointed is identified 384 via the f_pos variable. Iff_pos identifies a page with data 392, the checkpoint is created. Thecheckpoint is created by copying said page to user space, setting thevariable nBytes to the number of bytes transferred, and updating thef_pos value to designate the next available page. f_pos is set to thevalue of vm_end for the last page, and to address of the next page forall other pages. The check 392 to determine if this page contains datais thus accomplished by comparing the f_pos value to the vm_end value ofthe last page. In all cases, the number of bytes transferred to userspace is returned by the read( ) function.

This just disclosed read( ) function requires a more sophisticateduser-space caller to accommodate the different interpretation of thereturn values

-   -   unsigned byte ubuff[PAGE_SIZE];    -   int fd;    -   //set/seek to first page.    -   currPageAddr=lseek(fd,0,SEEK_SET); // set file pos to zero    -   boolean done=false;    -   while (!done)    -   {        -   count=read(fd,ubuff,PAGE_SIZE)        -   if (count>0)        -   {            -   Save the page data (ubuff) for currPagrAddr;            -   //save addr of next page to be read            -   currPageAddr=lseek(fd,0,SEEK_CUR);        -   }        -   else        -   {            -   done=true; // at the end of a checkpoint;        -   }    -   }

In continuation of section 3 an example embodiment, without errorhandling, of the seek( ) for /dev/ckpt is

-   -   static loff_t ckpt_seek(struct file*filp,        -   loff_t offset, int type)    -   {        -   loff_t newPos=*f_pos;        -   switch (type)        -   {            -   case SEEK_SET: newPos=offset; break;            -   case SEEK_CUR: newPos=filp→f_pos+offset;                -   break;            -   case SEEK_END: newPos=SIZE−offset;                -   break;            -   Default:                -   newPos=−EINVAL;        -   }        -   filp→f_pos=newPos;        -   return newPos;    -   }

8.1 Further Optimizations Using PTEs

The checkpointing process may be further optimized by keeping track ofwhich pages have been modified between successive checkpoints. Referringto FIG. 12 for illustrative purposes: The first checkpoint takenincludes page 1 360, page 2 362 and page 3 364. By way of example, atthe time of the second checkpoint data in pages 1 and 2 is unchanged,while the data in page 3 has changed. The only page that needs to beincluded is thus page 3, as pages 1 and 2 already are known and storedin an earlier checkpoint. The technique where only changed pages arecheckpointed is called incremental checkpointing. As long as theprocessing it takes to keep track of the changes in the pages is lessthan the time it takes to checkpoint all pages, incrementalcheckpointing offers a performance improvement. If checkpointing is donefrequently, the number of pages changed will generally be much lowerthan the total number of pages, and incremental checkpointing is thus animprovement.

By way of example, the Linux operating system keeps track of page accessusing the _PAGE_ACCESSED and _PAGE_DIRTY bits as previously disclosed insection 6. The bits are generally automatically set by the CPU hardwareas the program is executing. The present invention repurposes the_PAGE_DIRTY bit in the following way:

After the completion of a checkpoint, the current _PAGE_DIRTY is copiedinto one of the unused bits in the page table entry and thereby savedfor future reference. In the following disclosures _PAGE_DIRTY _COPY isused as a short hand for said copy of the _PAGE_DIRTY bit at the time ofthe most recent checkpoint. As the _PAGE_DIRTY bit is used by severallow-level functions of the kernel, said low-level functions are updatedto use _PAGE_DIRTY _COPY instead of the original _PAGE_DIRTY bit. Thebenefit is thus that the _PAGE_DIRTY bit, which is automatically set bythe CPU hardware, becomes the tracking mechanism for which pages need tobe included in the next incremental checkpoint, and that no further datastructures are necessary in order to maintain complete tracking ofchanged pages. Upon completing the checkpoint, and after copying_PAGE_DIRTY to _PAGE_DIRTY _COPY, the _PAGE_DIRTY bit is cleared.

FIG. 14 illustrates by way of example embodiment 400 the disclosuresabove in further detail and their placement within the context of theearlier teachings. Referring to FIG. 13 for illustrative purposes, thecore checkpointer (394, 396, 398) is activated via blocks 392 and 388.FIG. 14 includes 402 said two blocks 392 and 398 from FIG. 13 toillustrate the changes to the previous teachings. Instead ofcheckpointing every page, as illustrated on FIG. 13, the incrementalcheckpointer operates as follows:

First check 404 is to determine if this is the first checkpoint. If itis the first checkpoint, there is no prior checkpoint and thus no_PAGE_DIRTY reference available. The checkpoint checkpoints the currentpage 406, copies 408 the _PAGE_DIRTY bit into _PAGE_DIRTY _COPY andclears 410 the _PAGE_DIRTY bit. Finally, the nBytes is set 412 to thenumber of bytes transferred in copy_to _user( ) as previously disclosed.If this is the second or later checkpoint 404 the page table entry flagis checked to see if the _PAGE_DIRTY bit is set. If _PAGE_DIRTY is set,the page has been modified and needs to be checkpointed, which isaccomplished with the just described set of steps 406, 408, 410 and 412.If the _PAGE_DIRTY is not set, the data on the page is unmodified andthe page does not need to be checkpointed. The number of bytes, nBytes,is thus set 416 to zero. Following setting the number of bytestransferred 412, 416 the current file pointer is updated 417 to point tothe start of the next page or the end of the last page, as previouslydisclosed. Finally, the checkpointer exits 418 as the current page hasbeen processed.

The just-disclosed incremental checkpointing offers several improvementsover Neary and Backensto: Where Neary and Backensto marked pages“read-only” and rely on a segment fault to capture changed pages; thepresent invention relies exclusively on the PAGE_DIRTY bit, which isautomatically updated by the CPU. In Neary, by primarily addressing theincremental checkpointing from user space, each segment fault causes atransition from kernel-space to use-space to handle the errorconditions. Said transition is generally expensive and slows applicationexecution. The present invention exclusively deals with the access issuein kernel space and requires no page fault handling.

8.2 Merging Full and Incremental Checkpoints

When using incremental checkpointing, the incremental checkpoints aremerged into a full checkpoint. The checkpointing process starts with onefull checkpoint, i.e. a checkpoint that includes all process memorypages, and is followed by checkpoints where only the changed pages areincluded. A new incremental checkpoint contains only those pages thathave changed since the last checkpoint, and the update to the fullcheckpoint proceeds as follows: every page in the incremental checkpointis written to the full checkpoint, thereby updating the checkpoint withthe most recent checkpoint data. The process of merging an incrementalcheckpoint into a full checkpoint is called merging throughout thedisclosures of the present invention.

By way of example, if an incremental checkpoint contains page 300, thepage 300 in the full checkpoint is updated. If the full checkpointcontains a page 301, and the incremental checkpoint does not containpage 301, said page 301 in the full checkpoint is left unchanged.

Upon the completion of a checkpoint merge, the full checkpoint isup-to-date with the latest incremental changes and may be used as acheckpoint for a full process restore. Incremental checkpoints on theirown are incomplete, as they only contain changed pages, and thereforecannot directly be used for a process restore.

FIG. 15 illustrates by way of example embodiment 440, the control flowof the user-space side of incremental checkpointing. As disclosed above,checkpointing starts by seeking to the beginning of the address to becheckpointed 442. If there are no more data to be collected 444,checkpointing is complete 458, and the checkpointing cycle isterminated. If there is more checkpointing data to collect 446, it isfirst determined 448 if this is a full checkpoint or an incrementalcheckpoint. If it's a full checkpoint, the checkpoint is written tostorage 452. If it is an incremental checkpoint, the incrementalcheckpoint is merged 450 into the current full checkpoint, and said fullcheckpoint written to storage 452. After writing the full checkpoint452, the checkpointer waits for the next checkpoint trigger 456. Whenthe checkpoint is triggered the cycle repeats 444. From the user-spacepart of checkpointing, the incremental checkpointer only requires oneadditional step of merging 450 incremental pages into a full checkpoint.

8.3 Storing Checkpoints and Compression

Each time an incremental checkpoint is merged into the previouscheckpoint a new full checkpoint is formed. It differs from the previouscheckpoint by the pages in the incremental checkpoint. Said fullcheckpoints are written to memory, local storage, networked store orother remote storage for potential use later. The application may befully restored from a full checkpoint, as disclosed in the followingsection.

In a preferred embodiment, the checkpoints are written to memory orstorage uncompressed; in an alternate preferred embodiment thecheckpoints are compressed prior to being written to storage.Conversely, upon reading a compressed checkpoint during restore, thecheckpoint is decompressed.

8.4 Deletion of Pages

As an application executes it allocates and de-allocates memory. As theapplication allocates memory it is automatically reflected in the VMAsas additional pages and is thus automatically picked up by the disclosedincremental checkpointer.

When an application de-allocates memory the operating systems andlibraries remove the allocation from the application process that hadallocated the memory. Depending on the size of the de-allocate memoryand the memory allocator, the corresponding memory page may bede-allocated and removed from the virtual memory area. When and ifmemory is returned to the operating system is not predictable, and thepresent invention addresses the issue of de-allocating memory asfollows:

As previously disclosed, the checkpointer starts by taking a fullcheckpoint, which contains a complete memory image and then takes one ormore incremental checkpoints. After a set number of incrementalcheckpoints, a full checkpoint is taken again, and the cycle restarts.The full checkpoints fully account for allocated and de-allocated memoryand every full checkpoint thus ensures that de-allocated memory has beenproperly accounted for in the checkpoint.

9. Restoring from Checkpoint

FIG. 9 illustrates by way of example embodiment 280 the generalprocesses by which an application is restored from a checkpoint. First,the initial process 282 is created. The initial process is responsiblefor reading the checkpoint and creating the application processes. Theinitial process first reads the global checkpoint data 284 including theAVS. The AVS is required before creating any of the applicationprocesses, as the virtualization subsystem may be queried by theindividual component processes. This is followed by reading the processhierarchy 286 and the data shared between application processes 288. Atthis point the initial process is able to recreate the individualprocesses that comprise 290 the application as it has loaded the processhierarchy. The individual component processes are then created, thecheckpoints overlaid, interceptors installed and state rebuild. Theprocess rebuilding is first performed for the first process in thehierarchy P1 292, followed by the second process P2 294, and eventuallyfinishes with the last process Pn 296. When, by way of example, processP1 is created, checkpoint loaded, interceptors installed and staterebuilt, it is done within the process hierarchy 290 as provided in thecheckpoint. It is obvious to anyone with ordinary skills in the art thatthe above disclosures naturally extends to application comprised of anynumber processes with any hierarchy. When all processes have beencreated and initialized, the resources identified within the AVS areremapped 298 to ensure that the image is correctly mapped on the newhost OS. Finally, with the entire state recreated, the application isreleased 299 from the barrier and resumes running from the place of thecheckpoint.

The restoration of individual process checkpoints are now disclosed infurther detail. FIG. 8 illustrates by way of example embodiment, 240 thestructure of the checkpoint. Specifically, the layout of the checkpointfor an individual processes is given in blocks 260-272. During arestoration of a checkpoint, each page in the checkpoint is read and thedata written to the appropriate memory page. By way of example the data264 from the first page in the checkpoint 262 is written to thecorresponding page in application memory. The same applies for all otherpages within the address space of the application. The page info blockis used to update the page table and VMA page flags as appropriate. Thedetailed teachers of restoring VMA and PTE are given below.

As disclosed above, the kernel uses Virtual Memory Areas (VMAs) to keeptrack of memory. Part of restoring an application's memory image from acheckpoint therefore requires re-generation of the VMA data structures.Specifically, the vm_area_struct embedded within the mm_struct must berebuilt to match the memory pages included within the checkpoint.

To restore from a checkpoint requires said checkpoint to be a fullcheckpoint, as opposed to an incremental checkpoint. Since a fullcheckpoint contains all memory pages for the process, rebuilding the VMAinfrastructure is comprised of first deleting all VMA entries in thevm_area _struct followed by adding each page one at a time.

9.1 Restoring VMAs and PTEs.

The virtual memory areas are process private and can thus be rebuilt tomatch the checkpoint exactly. The page table and PTEs are part of theunderlying operating system and must therefore be treated differentlythan process-private structures.

In the preferred embodiment, restoring a page from a checkpoint isperformed as follows: First a vm_area _struct is allocated to representthe page, the page is allocated with do_mmap( ), the vm_area _struct isupdated to match the just allocated page, the vm_area _struct attributesare updated with the attributes stored in the page info block in thecheckpoint, and the vm_area _struct is merged into the VMAinfrastructure for the process using vma merge( ). The just-disclosedembodiment thus restores a page from the checkpoint to the same virtualaddress it occupied within the address space of the original processwhere the checkpoint originated. Associated with the page in theprocess' address space is the physical PTE and its associated protectionand state. Using the traversal algorithm disclosed previously, the PTEis identified for each page, and said PTE is updated with the PTEattributes stored in the page info block in the checkpoint of said page.

The entire process is restored by first clearing all vm_area_structsfrom the mm_struct, and then restoring each page as just disclosed. Therepeated restoration of every page in the checkpoint rebuilds theprocess' memory image to match the image at the time of the checkpoint,and by also updating the associated physical PTE, the operating system'sview and the process' internal state is reconstructed to match theoriginal process.

The kernel functions to manipulate VMAs and PTEs are well documented andwill therefore not be further described herein.

9.2 Remapping AVS Entries.

Finally, after all processes have been recreated and restored, theresource information in the AVS is remapped against the new image.Resources, such as open files, have components both in user-space and inkernel-space, and must be rebuilt to be functional after a restore. Byway of example, the file is re-opened from user space, which triggers arecreation of its kernel state. This last step updates the internals ofthe resource and makes is a functional resource on the new hostoperating system.

After remapping of AVS entries the application is ready to run.

10. Deployment Scenarios

FIG. 10 illustrates by way of example embodiment 300 a variety of waysthe invention can be configured to operate.

In one embodiment, the invention is configured with a central fileserver 302, primary server 304 and backup server 306. The primary server304 runs the primary application and the backup serves as backup. Theprimary 304 and backup 306 are connected to each other and the storagedevice 302 via a network 308. The network is connected to the internet316 for external access. In another embodiment the primary server 304has two backup servers; backup 306 and backup-2 305. In yet anotherembodiment the primary 304 runs in the data center, while the backup 317runs off site, accessed over the internet

In one embodiment a PC client 312 on the local network 308 is connectedto the primary application while the backup application is prepared totake over in the event of a fault. In another embodiment a PC 314 isconfigured to access the primary application server 304 over the publicinternet 316. In a third embodiment a cell phone or PDA 310 is accessingthe primary application 304 over wireless internet 316, 318. The presentinvention is configured to server all clients simultaneouslyindependently of how they connect into the application server; and inall cases the backup server is prepared to take over in the event of afault

Finally, as the interceptors and kernel module are componentsimplemented outside the application, the operating system and systemlibraries, the present invention provides checkpointing withoutrequiring any modifications to the application, operating system andsystem libraries.

The just illustrated example embodiments should not be construed aslimiting the scope of the invention but as merely providingillustrations of some of the exemplary embodiments of this invention

11. Conclusion

In the embodiments described herein, an example programming environment,systems and configurations were disclosed for which one or moreembodiments according to the invention were taught. It should beappreciated that the present invention can be implemented by one ofordinary skill in the art using different program organizations andstructures, different data structures, different configurations,different systems, and of course any desired naming conventions withoutdeparting from the teachings herein. In addition, the invention can beported, or otherwise configured for, use across a wide-range ofoperating system environments.

Although the description above contains many details, these should notbe construed as limiting the scope of the invention but as merelyproviding illustrations of some of the exemplary embodiments of thisinvention. Therefore, it will be appreciated that the scope of thepresent invention fully encompasses other embodiments which may becomeobvious to those skilled in the art, and that the scope of the presentinvention is accordingly to be limited by nothing other than theappended claims, in which reference to an element in the singular is notintended to mean “one and only one” unless explicitly so stated, butrather “one or more.” All structural and functional equivalents to theelements of the above-described preferred embodiment that are known tothose of ordinary skill in the art are expressly incorporated herein byreference and are intended to be encompassed by the present claims.Moreover, it is not necessary for a device or method to address each andevery problem sought to be solved by the present invention, for it to beencompassed by the present claims. Furthermore, no element, component,or method step in the present disclosure is intended to be dedicated tothe public regardless of whether the element, component, or method stepis explicitly recited in the claims. No claim element herein is to beconstrued under the provisions of 35 U.S.C. 112, sixth paragraph, unlessthe element is expressly recited using the phrase “means for.”

What is claimed is:
 1. A system, comprising: one or more CentralProcessing Units (CPUs) operatively connected to computer system memoryand configured to execute one or more multi-process applications on ahost with a host operating system; a kernel-space checkpointerconfigured to provide incremental kernel-space checkpointing of themulti-process applications; and one or more user-space interceptorspreloaded into an address space of each application process, whereinsaid user-space interceptors comprise a barrier; wherein saidkernel-space checkpointer is called while the execution of applicationprocesses are halted at said barrier, and said kernel-space checkpointerincrementally checkpoints the processes of the one or more multi-processapplications; wherein said incremental checkpointing is comprised oftracking dirty pages for page table entry (PTE) pages within the addressspace of the one or more multi-process applications; wherein a dirtypage bit is repurposed to track dirty pages for incrementalcheckpointing.
 2. The system according to claim 1 wherein said hostoperating system is one of Windows, Linux, Solaris, Android, MacOS, iOS,or UNIX.
 3. The system according to claim 1 wherein said kernel-spacecheckpointer is implemented as one of a kernel module, a loadable kernelmodule, or kernel loadable module.
 4. The system according to claim 1wherein said kernel-space checkpointer is compiled into the kernel. 5.The system according to claim 1 wherein said kernel-space checkpointeris implemented as one of a character device or block device.
 6. Thesystem according to claim 1, comprising at least one page tableconfigured to map between process virtual addresses and physical memoryaddresses.
 7. The system according to claim 1, wherein said user-spaceinterceptors comprise creation of a per-process checkpointing thread andan application virtualization space providing a private resource namespace.
 8. The system according to claim 1, wherein a page table entrywith page dirty set is included in an incremental checkpoint, and a pagetable entry with page dirty bit cleared is excluded from the incrementalcheckpoint.
 9. The system according to claim 8, wherein the page tableentry dirty bit is cleared after taking an incremental checkpoint. 10.The system according to claim 1, wherein virtual memory areas (VMAs) ofthe one or more multi-process applications are used to identify memorypages for multi-process applications.
 11. The system according to claim10, wherein for each virtual memory page identified by said virtualmemory areas, a page table is traversed and a corresponding page tableentry is identified.
 12. The system according to claim 10, whereincheckpoint information for each page contains virtual memory area pageinformation and corresponding page table entry information.
 13. Thesystem according to claim 10, wherein said memory pages are identifiedby traversing virtual memory areas (VMAs) for each application process,and for each virtual memory area identify pages comprising each virtualmemory area.
 14. The system according to claim 1, further comprising acheckpointer configured to: first taking one full checkpoint comprisingmemory pages in multi-process applications, and clearing all page dirtybits for said memory pages; taking one or more incremental checkpoints,and merging said incremental checkpoints into said full checkpoint;wherein pages included in said incremental checkpoints replace saidpages in the full checkpoint, and other pages in said full checkpointremain unmodified.
 15. The system according to claim 14, wherein saidmerged checkpoints are written to one of memory, local storage, networkstorage, or remote storage.
 16. The system according to claim 15,further comprising compressing-said checkpoints before writing to one ofmemory, local storage, network storage, or remote storage.
 17. Thesystem according to claim 14, wherein: a full checkpoint is comprised ofglobal state for application processes, process hierarchy, shared state,and individual process checkpoints; and said individual processcheckpoints are comprised of one or more page information blockscomprised of virtual memory areas page information and page table entrypage information.
 18. The system according to claim 14, wherein a pagein said full checkpoint is left unmodified if said incrementalcheckpoints does not contain said page.
 19. The system according toclaim 1, wherein said incremental checkpoints are transferred fromkernel-space to user-spacing using one of copy to user( ) or memorymapped pages.
 20. A method, comprising: executing one or moremulti-process applications on a host with a host operating system;providing kernel-space checkpointing of the multi-process applicationsvia a kernel-space checkpointer, wherein one or more user-spaceinterceptors are located in an address space of each applicationprocess, wherein the one or more user-space interceptors comprise abarrier; calling the kernel-space checkpointer while the execution ofapplication processes are halted at said barrier; incrementallycheckpointing, by the kernel-space checkpointer, the processes of themulti-process applications; wherein said incremental checkpointing iscomprised of tracking dirty pages for page table entry pages within theaddress space of said multi-process applications; wherein a dirty pagebit is repurposed to track dirty pages for incremental checkpointing.